


RWJ*0? 



CMC REPORT RESUME 




EO OIO 20t I-SI -At 24 tREVI 

RECENT DEVELOPMENTS IN ORTA COLLECTION ANO ANALYSIS NCR EXPERIMENTS* 
RAGSDALE. RONALD 

T 02 A 9143 UNIVERSITY OP PITTSBURGH. LEARNtNO R ANO 0 CTA.« PA. 
AR-S-CMJ-V 
«*PID»G0 

EORS PRICE NP-S0.09 HC-90.S* 14P. 

’COMPUTERS. ’COMPUTER PROGRAMS. ’GAYA COLLECTION. ’DATA ANALYSIS. 
EXPERIMENTAL PROGRAMS. P1TTSEUR0H. PENNSYLVANIA 

THIS MORNING PAPER OPPERS A RE ISP DESCRIPTION OP THE UTILIZATION OF 
CONPUTERS AS CONTROL DEVICES. LISTING GOTH THEIR ADVANTAGES ANO 
DISADVANTAGES. SUCH OTHER APPLICATIONS AS SIMULATION ANO *OATA 
PROCESSING* MERE NOT COVERED IN THIS DOCUMENT. (LPI 



U S DEPARTMENT OF HEALTH, EDUCATION ANO WELFARE 

Office of Education 

TO, document t«, opinion. 

Z2Z mp.ooe,.« «*» Of*. of EducaUon 

nosition or policy. 



RECENT DEVELOPMENTS IN DATA COLLECTION AND 
ANALYSIS FOR EXFERItCNTS 



Ronald Ragsdale 

£dC/6X»7 

Learning Research and Development Center 
University of Pittsburgh 



February, 1966 



the research and development reported herein teas performed pursuant 
to a contract with the United States Office of Education* Department of 
Health* education* and Wei fare under the provisions of the Cooperative 
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Recent Developments in Data Collection and 
Analysis for Experiments 

Whenever a symposium is organized) a set of titles is selected in such 
a v/ay that the unifying theme of the several presentations is readily ap- 
parent • Then each participant faces the task of skillfully using his title 
as an introduction to the topic he is willing to talk about) with the added 
restriction that he try to avoid overlap with the other papers, which he 
has not seen. 

The original title of this pax^er was ’Recent developments in data proces- 
sing” and the combination of this title and the present title probably gives 
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a much clearer* i A the topic I have in mind than either of them does by 

* 

itself. More specifically , I plan to discuss the role of the computer in ex- 
periments as a device to control conditions, collect data and make analysis 
relevant to the ongoing control process. 

Before getting into this more recent computer application, let me 
briefly mention applications which I will not discuss. Host of these can 
be broadly classified under the heading of "data processing* ** I will not 
talk about any statistical data processing in which a person presents data 
to the computer to be analyzed, summarized , collated, formatted, etc., or 
record keeping in which new data is added to some previously accumulated. 

The only recent development 1 shall mention in this area concerns a mythical 
(?) computer which bears the acronym GIGO. My colleague, Henry Hausdorff, 
who conceived of the machine, explains that the initials stand for ’Garbage 
In, Garbage Out. 1 ’ 
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Another well known area of computer application which will not be dis- 
cussed is that of simulation* r J T e simulation may be of another computer « as 
in digital simulation of analog computers , but we are generally more interes- 
ted in the simulation of a person or group. This type of application can 
lead to insights into the processes involved in human behavior. Further , it 
is invaluable in determin \g the consequences of a particular mathematical 
model. Recent developments in this area* however* are more the accomplish- 
ments of individual researchers than of any developments in research methodo- 
logy or the tools available. 

The Computer as a Control Device 

For several years the small or medium-sized computer has been uc«d in 
industry to control some process by continuously adjusting various parameters. 
These process control systems are applied most frequently to continuous pro- 
cesses such 99 rolling mills * steel sa^dag. batch blending- Typically ; 
the variables involved are so fast-changing and so interrelated that it is 
impossible for an unaided operator to know exactly how to react to process 

disturbances On the cither hand* a computer system can perform an entire 

% 

spectrum of functions ranging from simple data logging* in which it irerely 
reports variables* to complete process control. Between these extremes lie 
various mixes of logging* supervision* and control. 

In recent years* the behavioral sciences have evolved several parallel 
computer applications. That is the computer has been used to carry out ex- 
periments* collect the data* and ocassionally alter the experimental pro- 
cedures on the basis of the collected data. The analogy of this application 
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to that of industrial pro :e 3a control should be even clearer when one consi- 
ders the general area of computer-assisted instruction* In the latter case 
it is the learning process which is being monitored and parameters are varied 
on the basis of the students responses* 

Present Computer-Controlled Experiments 

Many of the experiments now being done in computer-assisted laboratories 
do not really take advantage of the computers capabilities* In fact* it is 
not an easy task to think of an experiment which can be controlled by a 
computer and cannot be duplicated with conventional experimental apparatus* 
The difficulty of this task may be due to rigidity in thinking brought on by 
traditional courses in research methodology* The primary advantage of the 
computer lies in its ability to modify the procedure on the basis of the 
collected data, but very few current experiments make use of this feature* 

The ability of the comouter to collect ljumtre Mnouatg of dbta in short 
periods of time is one advantage currently being exploited! although mainly 
in nuclear physios experiments* It is also possible to put this ability to 
use in behavioral science applications* It is a simple matter for a compu- 
ter to simultaneously collect data on latency of response! type of response, 
amplitude of response 1 physiological variables! etc*, all with a high degree 
of accuracy and for more than one subject* This oould be a disadvantage if 
the experimenter gathers data for a large number of variables just to see if 
anything comes out statistically significant* The saving grace for computers 
is that although such multiple data collection is technically possible! it 
is initially not economically desirable, if only because of the extra 
ming involved* 
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Computers have also been used quite extensively in the generation of 
stimulus materials. This is particularly appropriate when an oscilloscope 
is used for display. Given the dimensions on which stimuli can vary it is 
usorlly possible to write a program to generate all possible stimulus 
objects. In this manner the experimenter can have an extremely large pool 
of potential stimulus objects even though a large number of them may never 
be used. 

A further advantage for the use of computers in conducting experiments 
is timing flexibility. It is possible for the experimenter to specify with 
a high degree of accuracy , the timing of all experimental events under his 
control. The flexibility of the computer allows the timing to be easily 
varied over trials, or to be conditional on the occurrence cf other events. 
Some Disadvantages of Computer Control 

As long a a I have listed some advantages of computer control) it is 
only fair that I also list some disadvantages. The most serious disadvan- 
tage is probably the limited number of response modes available. In many 
cases there is only one response jode « that of depressing a key on a key- 
board. If facilities are available) one can adapt more sophisticated resp- 
onse devices for computer input, but for the typical experimenter the 
availability is limited. This situation is being alleviated as new response 

devices become more common. Devices recently developed, or under develop- 
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ment, allow the computer to sense the movement of an electronic pencil, the 
location of a finger touch and the movement of blocks. 

A further disadvantage of computer-assisted experimentation is the 
length of time it takes before an experiment can be implemented. 1 am re- 
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f erring to the time necessary for computer programming and debugging. Un- 
leee one plans to process a large number of subjects, or do a series of 
related experiments u s ing the same general procedure, it may be more expe- 
ditious to carry out the experiment by conventional means . Once again, 
this is an area where steps are being taken to bring relief. In this case 
relief is in the form of special computer languages. Several la ng ua g es for 
the presentation of instructional materials have reached different stages 
of development. A language for the acquisition of oceanographic data has 
also been created and hopes for a general experimental language do not seem 
out of place. 

One disadvantage that holds for almost any computer program that is 
at all complex, is that one can never really be sure that a program will 
work for all possible cases until they have all been tried. is pro- 

bably more serious for experimentation than for analysis programs since there 
ere usually several independently derived analysis programs to serve as 
checks on each other. Nevertheless, some analysis routines have been faclty 
and yet been used for long periods of time. It will be quite dif ficult to 
determine if the routines which obtain the data are in error. 

Bauinment-ghgrd wg vg , Time-flH»*H 

In general , experiments run under computer control, particularly those 
using small computers, are done in an equipment-sharing manner* Different 
experiments are min at different times, with each being loaded into the 
computer as it is needed* In this mode of operation, each experimental pro- 
cedure may use any or all of the available equipment. 



An alternative to equipment-sharing is time-sharing , in which several 
experiments are conducted in 1 particular time period using a single comp- 
uter* This is feasible since in a typical experiment the computer can 
spend over 90# of its time waiting for the subject to respond* The imple- 
mentation of a time-sharing eaqperimental laboratory requires a greater ini- 
tial investment in programming but allows more efficient use of the major 
component of the laboratory! the computer. The computer which most readily 
adepts to this type of use is a small or medium sized computer in the 
$ 90,000 - $190,000 price range* A high input-output rate is desirable but 
a large amount of fast-access memory and great processing capability are 
not essential* A slower mass memory is impoi*tant, however, for servicing 
even a smell number of stations with any flexibility* This slower mass 
memory can be for storing data, both current and background, decision 
and analysis programs, subject matter, and general service routines* 
Implications for Data Analysis 

There are implications that computerized data collection has for data 
analysis which are worth considering* I have already mentioned the relative 
ease with which large numbers of variables may be sampled. A feature worth 
considering in more detail is the possibility of altering the experimental 
procedure, or instructional sequence, on the basis of the analysis of pre- 
vious responses* 

Let me give some examples of situations which might benefit from this 
facility • X have been _ formed that a reasonable hypothesis in paired- 
associate learning is that the learning of a particular pair is facilitated 



in proportion to the number of times it follows a correct response* This 
is a very difficult hypothesis to test unless you can vary the order of 
the list after each response* If the list order can be varied after each 
response! it is possible to randomly assign pairs to conditions of following 
a high percentage of correct responses and a low percentage of correct re- 
sponses* Using a computer for list presentation and data collection makes* 
the task quite simple. 

here are many situ.? t ions where a responsive display is desirable. 

This might involve plotting an equation! showing the output of an electri- 
cal circuit f etc* This can be viewed as a type of data analysis in that an 
analysis of the student's response uniquely determines the new stimuli's. 

The new stimulus for the student! i.e., the waveform of the circuit's output , 
will be a function of the students response! i.e. $ the values he gave for 
resistors! capacitors! etc. 

An exampl e that is even more applicable for the area of educational 
research is the investigation of the effect of subject matter parameters 
on learning rate. In this type of study one varies the mode and maraer 
of presentation, content variables, etc., in an attempt to optimize the 
learning rate for a subject. A problem in this area of research is that 
there are many effects which are not under the experimenters control, such 
as individual differences in students and time trends due to practice, 
fatigue, boredom and the like. If we could do away with some of these un- 
desirable sources of variance, this could be viewed as a problem for respo ns e— 
surface methodology. 
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A tool that may he useful in computer presentation of subject matter, 
which has not been extensively investigated, is latency of response* In 
a drill situation one often uses N consecutive correct as a criterion for 
success* It seems reasonable that one could terminate the drill earlier if 
the latency of the correct responses were used m an additional indication 
of learning* An example may make this point more clearly* A spelling drill 
which we use for demonstration purposes in our computer-assisted laboratory, 
requires that a subject spell a word correctly more times than he spells it 
incorrectly before the word is dropped from the drill* In spelling a word 
like SCIENCE a subject tey stumble on the I-E rule several times before get- 
ting the word right* Without getting into a debate on the merits of over- 
learning, I would like to suggest that instead of requiring the subject to 
spell SCIENCE correctly four or five times, we allow an alternative crite- 
rion of two correct responses with latencies well below his mean latency* 
Real-Time Analysis 

As experiments done under computer control ^become more prevalent, more 
attention will have to be paid to developing appropriate statistical tech- 
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niques for '^real-time " analysis, i.e*, analysis of the data while the ex- 
periment is in progress* Techniques which are described as '^sequential 
statistics" are probably not adequate to the task* Listed ur^er sequential 
statistics one finds measures of runs, auto-correlations and combinations 
thereof* It nay be that auto-correlation will be particularly helpful in 
studying variables such as motivation or attention, which may behave in a 
cyclical manner* fo this one might also add sequential designs such as 
fractional factorials, which are particularly helpful in exploratory studies, 



but an abundant supply of techniques is missing. 



Implementing real-time analysis techniques and using them for the modi* 
fication of experimental procedures will lead to what might be considered b; 
some to be another disadvantage of computer-assisted experimentation. The 
problem is that if an experimenter plans to make full use of the computer- 
assistance, he must consider all possible outcomes ior stage one of his 
experiment if he plans to modify the procedure foi stage two and must con- 
sider all possible outcomes of all possible variations of stage N if he 
plans to modify the procedure for stage N+l. This may be an overpowering 
restriction for some experimenters and they may choose to ignore some of 
the advantages of computer-assistance. For those of us who recommend this 
strange type of behavior, i.e., planning what to do v/ith the data before 
collecting it, computer controlled experiments may offer a blessing in dis- 
guise. 

In closing, let me say that although some people may see an advanced 
state of computer-assisted experimentation arriving very quickly, there are 
several inhibiting factors that I would like to mention. One involves an 
uncritical acceptance of computer accomplishment by those not familiar 
with computers. This is the ’!Did y j u see the computer that printed a picture 
of Santa and his reindeer? *• type comment. This attitude of awe is often 
fostered by computer users who sometimes attribute slightly supernatural 
properties to their programs (•'Chen depending on the type of student, the 
computer gives him • • ••) • This combination tends to lessen the demands that 
computer users be particularly creative in their applications. 
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A final inhibitor is the tendency for the use of a computer to become 
a goal rather than a means to an end. It is very easy to get so carried 
away with programming that one does nothing else. One extreme example of 
this might be a person who spends so much time creating a program to write 
up studies that he never has time to collect any data to write up. Hopefully} 
these inhibitors will have negligible effects and computer-assisted labo- 
ratories will be rapidly implemented. 
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